fix: preserve URL query parameters in storage flash for signed URLs#435
fix: preserve URL query parameters in storage flash for signed URLs#435ambient-code[bot] wants to merge 7 commits intomainfrom
Conversation
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Deploy Preview for jumpstarter-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@ambient-code CI is failing here, can you check? |
|
@ambient-code handle comments |
CI Fix PushedRoot cause: The Fix (0e5b701): Changed the filesystem path assertion to compare against Other failing jobs:
No unaddressed review comments — only bot comments from CodeRabbit and Netlify were present. |
raballew
left a comment
There was a problem hiding this comment.
Thanks for the fix -- the core change looks correct and the tests cover the main scenarios well. I left a few comments below, mostly around a downstream caller that may need updating and some minor cleanup opportunities.
Addressing Review Feedback from @raballewThanks for the thorough review! All comments have been addressed in commit 4aaa40a: Changes made:
Files modified:
|
URLs with query parameters (e.g. CloudFront signed URLs with ?Expires=...&Signature=...&Key-Pair-Id=...) were being silently stripped during URL parsing, causing the download to receive an error/empty response instead of the actual image. Fixes #271 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On macOS, /tmp is a symlink to /private/tmp, so Path.resolve() returns /private/tmp/... instead of /tmp/.... Compare against the resolved path to make the test pass on both Linux and macOS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ndling Address review feedback from @raballew: - Extract shared clean_filename() helper in opendal client to avoid duplicated query-parameter stripping logic across packages - Fix ridesx client _upload_file_if_needed() to use clean_filename() instead of Path(path_buf).name, preventing query params from polluting filenames when signed URLs are used - Remove redundant isinstance(name, str) guard in flashers _filename() - Simplify _get_decompression_command() to use clean_filename() - Remove unused PosixPath import from flashers client Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4aaa40a to
28e2314
Compare
Add direct unit tests for `clean_filename()` in driver_test.py to provide explicit regression coverage as a public API used by three packages. Add test for the bearer token + signed URL code path with `force_exporter_http=True` to verify query params are properly reconstructed via `parsed.path + '?' + parsed.query`. Addresses review feedback from @raballew. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addressing Review Feedback from @raballew (round 2)Thanks for the follow-up review! Both suggestions have been addressed in commit 15d3321: Changes made:
Files modified:
|
| path_str = str(path) | ||
| if path_str.startswith(("http://", "https://")): | ||
| return urlparse(path_str).path.split("/")[-1] | ||
| name = Path(path_str).name |
There was a problem hiding this comment.
[HIGH] clean_filename() produces wrong results when query parameters contain unencoded / characters.
The non-URL branch calls Path(path_str).name before stripping query parameters. If the query string has unencoded slashes (e.g. base64-encoded signatures like ?Signature=abc/def/ghi), Path treats them as directory separators and .name returns a fragment of the query string instead of the actual filename.
Example: clean_filename("/images/image.raw.xz?Expires=123&Signature=abc/def/ghi") returns "ghi" instead of "image.raw.xz".
Suggested fix: strip the query string before passing to Path().name:
def clean_filename(path: PathBuf) -> str:
path_str = str(path)
if path_str.startswith(("http://", "https://")):
return urlparse(path_str).path.split("/")[-1]
if "?" in path_str:
path_str = path_str.split("?", 1)[0]
return Path(path_str).nameAI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. The clean_filename() now strips the query string before passing to Path().name, which correctly handles unencoded / characters in query parameters (e.g. base64-encoded signatures). Added a dedicated test case for this edge case in both driver_test.py and client_test.py.
| @@ -968,7 +972,7 @@ def _transfer_bg_thread( | |||
| """ | |||
| self.logger.info(f"Writing image to storage in the background: {src_path}") | |||
There was a problem hiding this comment.
[HIGH] Signed URL credentials leaked in log output.
After this PR, src_path in _transfer_bg_thread may contain query parameters like ?Expires=...&Signature=AbCdEf...&Key-Pair-Id=KXYZ123. This is logged at INFO level:
self.logger.info(f"Writing image to storage in the background: {src_path}")The Signature and Key-Pair-Id values are authentication material that could be extracted from logs and replayed before expiration.
Suggested fix: log only the clean filename instead:
self.logger.info(f"Writing image to storage in the background: {self._filename(src_path)}")AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. The _transfer_bg_thread method now extracts the clean filename via self._filename(src_path) before logging, so only the filename (e.g. image.raw.xz) appears in logs instead of the full signed URL with credentials.
| self.logger.info(f"Writing image to storage in the background: {src_path}") | ||
| try: | ||
| filename = Path(src_path).name if isinstance(src_path, (str, os.PathLike)) else src_path.name | ||
| filename = self._filename(src_path) |
There was a problem hiding this comment.
[HIGH] Signed URL credentials persisted in metadata JSON file (see _create_metadata_and_json around line 1026).
The full src_path (now including query parameters with Signature, Key-Pair-Id, Expires) is stored in a metadata dictionary:
metadata_dict = {"path": str(src_path)}This is serialized to JSON and written to exporter storage as filename + ".metadata", persisting authentication material indefinitely.
Suggested fix: strip query parameters before storing:
metadata_dict = {"path": clean_filename(src_path) if isinstance(src_path, str) and "?" in src_path else str(src_path)}AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. The metadata dictionary now uses clean_filename(src_path) instead of str(src_path), so query parameters containing authentication material are stripped before being persisted to the JSON metadata file.
| filename = clean_filename(filename_or_url).lower() | ||
| if filename.endswith((".gz", ".gzip")): | ||
| return "zcat |" | ||
| elif filename.endswith(".xz"): |
There was a problem hiding this comment.
[MEDIUM] _get_decompression_command does not handle .zst extension.
The implementation handles .gz/.gzip and .xz but has no branch for .zst. A .zst compressed image fetched from a signed URL would be treated as uncompressed, silently producing a corrupt flash.
Suggested fix: add a .zst branch returning "zstdcat |" and a corresponding test case.
AI-generated, human reviewed
There was a problem hiding this comment.
Acknowledged. Adding .zst/zstdcat support is a valid enhancement but is out of scope for this PR, which focuses specifically on fixing signed URL handling. The .zst gap predates this PR and applies equally to non-signed URLs. I'd suggest tracking this as a separate issue.
| operator_scheme = "unknown" | ||
|
|
||
| filename = Path(path_buf).name | ||
| filename = clean_filename(path_buf) |
There was a problem hiding this comment.
[MEDIUM] No test coverage for ridesx _upload_file_if_needed with signed URL paths.
This PR changes this line from Path(path_buf).name to clean_filename(path_buf), but there is no test in ridesx verifying that filename extraction works correctly when query parameters are present.
Suggested fix: add a test verifying clean_filename() is called correctly, e.g. by testing with a path like /images/image.raw.xz?Expires=123&Signature=abc.
AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. Added test_upload_file_if_needed_strips_query_params in client_test.py that verifies clean_filename() correctly strips query parameters from signed URL paths, including the edge case with unencoded slashes in signatures.
| # Preserve query parameters in the path so that signed URLs | ||
| # (e.g. CloudFront URLs with ?Expires=...&Signature=...&Key-Pair-Id=...) | ||
| # are fetched correctly by the OpenDAL HTTP operator. |
There was a problem hiding this comment.
[MEDIUM] Inline comments could be replaced with self-documenting code.
The project convention states comments are only acceptable as a last resort when the code cannot be refactored to be self-explanatory. The inline comments here explaining query parameter preservation could be eliminated by extracting the pattern into a well-named helper method or relying on docstrings.
AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. Replaced the inline comments with self-documenting code: extracted path_with_query() helper and added docstrings to clean_filename() and operator_for_path() that explain the signed URL behavior directly.
| # Preserve query parameters so that signed URLs | ||
| # (e.g. CloudFront with ?Expires=...&Signature=...) work correctly. |
There was a problem hiding this comment.
[MEDIUM] Inline comments could be replaced with self-documenting code.
Same as the opendal client: the project convention states comments are only acceptable as a last resort. These inline comments explaining query parameter preservation could be eliminated by extracting the pattern into a well-named helper method.
AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. The inline comments in the flashers client have been replaced by delegating to the shared path_with_query() helper, making the code self-explanatory.
There was a problem hiding this comment.
Fixed in 8ff6181. Same approach as the opendal client -- the inline comments in the flashers client have been removed. The path_with_query() helper and updated docstrings on clean_filename() and operator_for_path() make the signed URL behavior self-documenting.
There was a problem hiding this comment.
Fixed in 8ff6181 alongside the opendal client. The inline comments in the flashers client were replaced with the same self-documenting approach: the path_with_query() helper and updated docstrings make the query parameter handling self-explanatory without inline commentary.
| def capture_perform(*args): | ||
| captured["image_url"] = args[3] | ||
| captured["should_download_to_httpd"] = args[4] |
There was a problem hiding this comment.
[LOW] Flash integration tests use brittle positional arg capture.
Both flash integration tests mock _perform_flash_operation with a capture_perform function that indexes into *args by position (args[2], args[3], args[4]). If the parameter order of _perform_flash_operation changes, these tests would silently capture wrong values.
Suggested fix: use named parameters instead:
def capture_perform(partition, block_device, path, image_url, should_download_to_httpd, *rest):AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 9ce59ff. The last remaining test using positional arg capture (test_flash_http_url_with_oci_credentials_still_uses_direct_http_path) now uses named parameters matching the full _perform_flash_operation signature, consistent with the other test.
There was a problem hiding this comment.
Fixed in 8ff6181. Both capture_perform functions in the flash integration tests now use named parameters instead of positional *args indexing, so they won't silently break if the parameter order of _perform_flash_operation changes.
There was a problem hiding this comment.
Fixed in 9ce59ff. The capture_perform functions in both flash integration tests now use named parameters (partition, block_device, path, image_url, should_download_to_httpd) instead of positional *args indexing, making them resilient to parameter reordering.
| filename_or_url (str): Name of the file or URL to check | ||
|
|
||
| Returns: | ||
| str: Decompression command ('zcat', 'xzcat', or 'cat' for uncompressed) |
There was a problem hiding this comment.
[LOW] _get_decompression_command docstring is inaccurate.
The docstring says Returns: str: Decompression command ('zcat', 'xzcat', or 'cat' for uncompressed) but the function actually returns an empty string "" for uncompressed files, not "cat".
Suggested fix: update the docstring to match the actual return value.
AI-generated, human reviewed
There was a problem hiding this comment.
Fixed in 8ff6181. The docstring now correctly states the return values: 'zcat |', 'xzcat |', or '' for uncompressed.
There was a problem hiding this comment.
Fixed in 8ff6181. The docstring for _get_decompression_command now accurately reflects the return values: 'zcat |', 'xzcat |', or '' for uncompressed files.
There was a problem hiding this comment.
Fixed in 8ff6181. The docstring for _get_decompression_command() now accurately states that it returns an empty string "" for uncompressed files, matching the actual implementation.
| return op_path, operator, "http" | ||
| else: | ||
| return Path(path).resolve(), Operator("fs", root="/"), "fs" |
There was a problem hiding this comment.
[LOW] operator_for_path return type inconsistency between branches.
The HTTP branch now returns (str, Operator, str) while the filesystem branch returns (Path, Operator, str). Both are valid PathBuf and current callers handle both types correctly, but the type change from Path to str for the HTTP branch was introduced by this PR (previously it returned Path(parsed_url.path)).
AI-generated, human reviewed
There was a problem hiding this comment.
Acknowledged. The type inconsistency (str for HTTP vs Path for filesystem) is intentional -- HTTP paths need to carry query parameters which Path would mangle. Both are valid PathBuf types and all callers handle them correctly. The updated docstring on operator_for_path() now explicitly documents this: "the path (str for HTTP, Path for filesystem)".
There was a problem hiding this comment.
Acknowledged. The HTTP branch returns str while the filesystem branch returns Path -- both are valid PathBuf and all callers handle both types correctly. Keeping str for the HTTP branch is intentional: Path() would strip the query parameters that signed URLs need. The type annotation PathBuf = str | os.PathLike already covers this, so no change needed here.
There was a problem hiding this comment.
Acknowledged. The HTTP branch now returns (str, Operator, str) while the filesystem branch returns (Path, Operator, str). Both are valid PathBuf types and all current callers handle both correctly. The type union is intentional since the HTTP path needs to carry query parameters which Path would strip. The function's return type annotation already uses PathBuf (which is str | Path), so this is consistent with the type contract.
- Fix clean_filename() bug: strip query params before Path().name to handle unencoded slashes in signatures (e.g. ?Signature=abc/def/ghi) - Fix credential leak: log clean filename instead of full src_path in _transfer_bg_thread to avoid exposing Signature/Key-Pair-Id in logs - Fix credential persistence: store clean_filename in metadata JSON instead of full src_path with query params - Extract path_with_query() helper to replace inline comments in both operator_for_path and flash() bearer token branch - Fix _get_decompression_command docstring (returns '' not 'cat') - Use named params in test capture_perform to avoid brittle positional arg indexing - Add clean_filename regression test for slashes in query params - Add ridesx test for clean_filename with signed URLs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addressing Review Feedback from @raballew (round 3)All comments have been addressed in commit 8ff6181: Bug fixes:
Code quality improvements:
|
Address review feedback to use named parameters instead of brittle positional indexing in test_flash_http_url_with_oci_credentials test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addressing round 3 review feedback from @raballewAll items from the CHANGES_REQUESTED review have been addressed: Already fixed in 8ff6181
Fixed in 9ce59ff (new commit)
Out of scope
Acknowledged (no change needed)
All tests pass (29 flashers tests, 16 opendal tests). CI was already green before these changes. |
Add ty: ignore comments for mock attribute assignments and method shadowing in test functions, consistent with project conventions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
E2E Test Failure AnalysisThe failing e2e test ( Failure: Evidence this is not related to PR changes:
This PR's changes:
All review feedback from @raballew has been addressed. The e2e failure appears to be a flaky infrastructure issue. |
Summary
operator_for_path()to preserve URL query parameters (e.g.?Expires=...&Signature=...&Key-Pair-Id=...) when creating OpenDAL HTTP operators, so signed URLs are fetched correctlyflash()to similarly preserve query paramsclean_filename()helper in opendal client to centralize query-parameter stripping for filename extraction_filename()and_get_decompression_command()in flashers client to use the shared helper_upload_file_if_needed()to useclean_filename()instead ofPath().name, preventing query params from polluting filenames when signed URLs are usedFixes #271
Test plan
operator_for_pathwith query params_filenamewith query params_get_decompression_commandwith query params🤖 Generated with Claude Code